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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address -- 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- ff the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )S Responsive to communication(s) filed on 07 August 2003 . 
2a)D This action is FINAL. 2b)M This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 1 1 , 453 O.G. 213. 
Disposition of Claims 

4) S Claim(s) 1 -33 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) S Claim(s) 1 -33 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)Q accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

1 1) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

1 3) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 

a)DAII b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 

Attachment(s) 

1) (3 Notice of References Cited (PTO-892) 4) d Interview Summary (PTO-413) Paper No(s). . 

2) d Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) O Notice of Informal Patent Application (PTO-1 52) 

3) □ Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) □ Other: 
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DETAILED ACTION 

1 . Response to communications filed on August 7, 2003. 

2. Claims 1 - 33 are pending in this office action. 

3. Applicants amend claims 1,18 and 24 

4. Applicant's arguments with respect to claims 1 - 33 have been considered but are 
moot in view of the new ground(s) of rejection. 

Allowable Subject Matter 

5. Claims 1 1 , 19 and 23 are objected to as being dependent upon a rejected base 
claim, but would be allowable if rewritten in independent form including all of the 
limitations of the base claim and any intervening claims. 

The followings is a statement of reasons for the indication of allowable subject 
matter: The prior art of record fails to teach and/or suggest "the step of examining the 
information stream further . . ., identifying a second set of documents that correspond to 
words found in the text; scoring the returned documents based on a plurality of criteria 
and ranking the documents based on their scores". 
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Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 

7. Claims 1, 2, 3, 5, 6, 9, 18, 24, 31 and 33 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over U.S. Patent 5,500,920 issued to Julian M. Kupiec 
(hereinafter "Kupiec") in view U.S. Patent 6,581 ,054 issued to Steven W. Bogrett 
(hereinafter "Bogrett"). 




Application/Control Number: 09/698,894 Page 4 

Art Unit: 2172 

Regarding claims 1, 18, 24 and 33, Kupiec teaches a method for providing 
collateral information for inclusion with an information stream, comprising steps of: 

examining the information stream to recognize a presence of events that occur in 
the information stream (see column 14, lines 15-20; "Kupiec discloses the phone 
sequence as information stream"). 

Kupiec does not explicitly teach automatically generating database queries from 
recognized events. 

analyzing results of said database queries so as to rank and select said results to 
be inserted into the information stream as information that is collateral to said organized 
events. 

Bogrett teaches automatically generating database queries from recognized 
events (see column 1 1 , lines 50 - 51 ). 

analyzing results of said database queries so as to rank and select said results to 
be inserted into the information stream as information that is collateral to said organized 
events (see column 7, lines 43 - 64). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Bogrett with the teaching of Kupiec wherein 
information that is collateral to said organized events are selected from the query 
results. Thus the selected information is analyzed and used to interpret users' concerns. 

Regarding claim 2, Kupiec teaches the step of analyzing comprises a step of 
ranking the database query results based on a plurality of criteria (see Fig. 7 step FL). 
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Regarding claim 3 f Kupiec teaches the plurality of criteria comprise a score 
derived from a free text search of the database using text that is automatically extracted 
from the information stream, on a number of named entities appearing in the text and in 
the database query results, and on a taxonomy path score, where the taxonomy path 
score represents an amount of relatedness between a taxonomy-related information 
element found in the text and a predetermined taxonomy tree (see column 5, lines 1 - 
30). 

Regarding claim 5, Bogrett teaches the step of examining comprises a step of 
automatically extracting text from the information stream (see column 7, lines 41 - 49). 

Regarding claim 6, Kupiec teaches segmenting the text into sentences (see 
column 24, lines 51 - 55); and 

operating on the sentences to identify topics that correspond to predetermined 
topic taxonomies, wherein the step of automatically generating database queries 
operates on identified topics (see column 16, lines 30 - 36). 

Regarding claim 9, Kupiec teaches the step of automatically extracting text from 
the information stream comprises a step of operating a character recognition system 
(see column 24, lines 32 - 35). 
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Regarding claim 31 , Bogrett teaches the database queries are automatically 
generated based on information corresponding to identified topics extracted from the 
audio/video stream, where the topics correspond to predetermined topic taxonomies 
(see column 1 1 , lines 50 - 51 ). 

8. Claims 4, 7, 8, 10, 13, 14, 20-22, 25, 26 -30 and 32 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Kupiec in view of Bogrett and further in view of 
U.S. Patent 5,835,667 issued to Howard D. Wactlar et al (hereinafter "Wactlar"). 

Regarding claim 4, Kupiec and Bogrett do not explicitly teach the database 
queries are automatically generated based on information corresponding to a list that 
identifies topics in text that is automatically extracted from the information stream, 
where the topics correspond to predetermined topic taxonomies. 

Wactlar teaches the database queries are automatically generated based on 
information corresponding to a list that identifies topics in text that is automatically 
extracted from the information stream, where the topics correspond to predetermined 
topic taxonomies (see column 15, lines 2 - 6). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Wactlar with the teaching of Kupiec and 
Bogrett wherein text are extracted sequences. The motivation is that these texts are 
used in searching desired audio/video. 
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Regarding claim 7, Wactlar teaches the step of automatically extracting text from 
the information stream comprises a step of operating a voice recognition system (see 
column 6, lines 57 - 58) 

Regarding claim 8, Wactlar teaches the step of automatically extracting text from the 
information stream comprises a step of extracting closed caption text (see column 7, 
lines 58-62). 

Regarding claim 10, Wactlar teaches the step of automatically extracting text from 
the information stream comprises a step of also generating text that is descriptive of a 
number of human faces that are present in an image conveyed by the information 
stream (see Figs. A-1 and A-2). 

Regarding claim 13, Kupiec the step of extracting comprises a step of ranking the 
extracted information based on a plurality of criteria, and where the step of multiplexing 
uses the ranked data (see column 12, lines 9 - 22). 

Regarding claim 14, Kupiec teaches the step of extracting comprises a step of 
ranking extracted document information based on a score derived from a free text 
search of a document database using the text, on a number of named entities extracted 
from the text that are found in the documents, and on a taxonomy path score, where the 
taxonomy path score represents an amount of relatedness between a taxonomy-related 
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information element identified in the text and a predetermined taxonomy tree (see 
column 12, lines 1 - 6). 

Regarding claim 20, Wactler teaches examining subsystem comprises at least one 
unit for automatically extracting text from the information stream, a unit for segmenting 
the text into sentences and at least one unit for operating on the sentences to identify 
topics that correspond to predetermined topic taxonomies, wherein said query 
generation subsystem automatically generates database queries based at least in part 
on identified topics (see column 12, lines 57 - 58). 

Regarding claim 21, Kupiec teaches text extracting unit comprises at least one of a 
voice recognition system, a system for extracting closed caption text, and a character 
recognition system (see column 24, lines 32 - 35). 

Regarding claim 22, Wactler teaches examining subsystem comprises a unit for 
generating text that is descriptive of a number of human faces that are present in an 
image conveyed by the information stream (see Figs.A-1 and A-2). 

Regarding claim 25, Wactlar teaches wherein further comprising a step of 
inserting the collateral information into the audio/video stream in real time or 
substantially real time (see column 6, lines 36 - 38). 



• • 1 
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Regarding claim 26, Wactler teaches examining includes a step of generating a 
speech transcript from at least the audio portion of the audio/visual stream, and wherein 
recognized events comprise speech topics (see column 6, lines 41 - 48). 

Regarding claim 27, Wactlar teaches the audio/video stream originates as a 
television broadcast signal (see column 1, lines 25 - 33). 

Regarding claim 28, Wactler teaches the audio/video stream originates at a meeting, 
and further comprising a step of presenting the collateral information to meeting 
participants in real time or substantially real time (see column 1, lines 14 - 18 and 30 - 
33). 

Regarding claim 29, Wactler teaches the step of presenting comprises a step of 
inserting the collateral information into the audio/video stream, and displaying the 
audio/video stream to the meeting participants (see Fig.4 step 42). 

Regarding claim 30, Wactler a step of archiving at least the collateral information 
(see column 6, lines 46 - 50). 

Regarding claim 32, Wactlar the step of examining includes steps of generating a 
speech transcript comprised of words from at least the audio portion of the audio/video 
stream; segmenting the words into sentences; and operating on the sentences to 
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identify topics that correspond to predetermined topic taxonomies, wherein the step of 
generating database queries operates on identified topics (see column 2, lines 44 - 53). 

9. Claims 12, 15, 16 and 17 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Wactler in view of Bogrett. 

Regarding claim 12, Wactler teaches a method for providing collateral information 
for multiplexing with an information stream, comprising steps of: 

converting the information stream into text (see column 6, lines 46 - 50); 

analyzing the text to identify information elements (see column 9, lines 42 - 47); 

extracting data from database search results that is relevant to the information 
stream (see column 15, lines 2 - 6); 

and multiplexing the data into the information stream for presentation at a destination 
of the information stream (see column 15, lines 20 - 26). 

Wactler does not explicitly teach automatically generating queries from the 
information elements for searching at least one database. 

Bogrett teaches teach automatically generating queries from the information 
elements for searching at least one database (see column 1 1 , lines 50 - 51 ); 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine teaching of Bogrett with teaching of Wactlar wherein the queries 
are used in selecting desire audio/video stream. The motivation is that automatically 
generating these data base query saves time. 
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Regarding claim 15, Wactlar teaches the queries are generated based on 
information elements that correspond to a list of information elements identifying topics 
in the text being analyzed, where the topics correspond to predetermined topic 
taxonomies (see column 6, line 67 and column 14, lines 46 - 57). 

Regarding claim 16, Wactlar teaches the step of analyzing the text comprises steps 
of segmenting the text into sentences and a step of operating on the sentences to 
identify topics that correspond to predetermined topic taxonomies, and wherein the step 
of automatically generating queries operates on identified topics (see column 12, lines 
57 - 58). 

Regarding claim 17, Wactlar teaches the step of analyzing the text comprises steps 
of at least segmenting the text into sentences, identifying names of entities within the 
text, and a step of operating on the sentences to identify topics that correspond to 
predetermined topic taxonomies, and wherein the step of automatically generating 
queries operates on identified topics and ranks the database search results based at 
least on numbers of named entities found and on an amount of relatedness between a 
taxonomy-related information element identified in the text and a predetermined 
taxonomy tree (see column 9, lines 41 - 56). 
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Conclusion 



10. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Fred I. Ehichioya whose telephone number is 703-305- 
8039. The examiner can normally be reached on M - F 8:00 AM to 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kim Y. Vu can be reached on 703-305-4393. The fax phone numbers for 
the organization where this application or proceeding is assigned are 703-746-7239 for 
regular communications and 703-746-7238 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is 703-303- 
3900. 



Fred Ehichioya 
Patent Examiner 
October 20,2003 




